Image processing including image correction

ABSTRACT

Long and short exposure time pixel information are input to pixel information. A long exposure time image set with the pixel values assuming all of the pixels have been exposed for a long time and a short exposure time image set with the pixel values assuming all of the pixels have been exposed for a short time are generated. A point spread function corresponding to the long exposure time image is computed as a long exposure time image PSF. A corrected image is generated using the short exposure time image, the long exposure time image, and the long exposure time image PSF. The corrected image is generated as a wide dynamic range image utilizing the pixel information for the long and short exposure time image. Utilizing the pixel information for the short exposure time image with little blurring, makes the corrected image a high quality corrected image with little blurring.

CROSS REFERENCES TO RELATED APPLICATIONS

This application is a continuation of and claims the benefit under 35U.S.C. §120 of U.S. patent application Ser. No. 13/452,977, titled“IMAGE PROCESSING INCLUDING IMAGE CORRECTION” and filed on Apr. 23,2012, which claims the benefit under 35 U.S.C. §119 of Japanese PatentApplication 2011-102915, filed on May 2, 2011, each of which is herebyincorporated by reference herein in its entirety.

BACKGROUND OF THE INVENTION

The present disclosure relates to an image processing device, an imageprocessing method, and a program. In particular, the present disclosurerelates to an image processing device, an image processing method, and aprogram that generate an image with a dynamic range that is wide.

A solid image capture element that is used in a video camera or adigital still camera, such as a CCD image sensor or a complementarymetal oxide semiconductor (CMOS) image sensor, accumulates an electricalcharge that corresponds to the amount of incident light and performs aphotoelectric conversion that outputs an electrical signal thatcorresponds to the accumulated electrical charge. However, there is anupper limit to the amount of the electrical charge that is accumulatedin the photoelectric conversion element, and when more than a fixedamount of light is received, the amount of the accumulated electricalcharge reaches a saturation level, such that what are called blown outhighlights, which are set at a saturated brightness level, occur inregions of the photographic subject where the brightness is greater thana fixed value.

In order to prevent this sort of phenomenon from occurring, processingis performed that adjusts the exposure time by controlling the periodduring which the electrical charge is accumulated in the photoelectricconversion element, in accordance with variations in the outside lightand the like, and that also adjusts the sensitivity to an optimum value.For example, for a bright subject, the exposure time is shortened byusing a faster shutter speed, shortening the period during which theelectrical charge is accumulated in the photoelectric conversionelement, and the electrical signal is output before the amount of theaccumulated electrical charge reaches the saturation level. This sort ofprocessing makes it possible to output an image that accuratelyreproduces the gray-scale levels of the subject.

However, in capturing an image of a subject that has a mixture of brightregions and dark regions, using a fast shutter speed means that theexposure time will not be sufficient for the dark portions, so thesignal-to-noise ratio worsens, and the image quality deteriorates. Inorder to accurately reproduce the brightness levels of the brightportions and the dark portions in a captured image of a subject that hasa mixture of bright regions and dark regions, processing must beperformed that increases the exposure time and achieves a highsignal-to-noise ratio for the image sensor pixels where the amount ofthe incident light is low and that avoids saturation in the pixels wherethe amount of the incident light is high.

As a technique for implementing this sort of processing, a technique isknown that sequentially captures and combines a plurality of images withdifferent exposure times. Specifically, an image with a long exposuretime and an image with a short exposure time are captured separately insequence. The technique generates a single image by performing combiningprocessing that uses the long exposure time image for the dark imageregions and uses the short exposure time image for the bright imageregions where the highlights are blown out in the long exposure timeimage. Combining a plurality of images with different exposures in thismanner makes it possible produce an image with a dynamic range that iswide and in which there are no blown out highlights, that is, a widedynamic range image (a high dynamic range (HDR) image).

For example, Japanese Patent Application Publication No. JP-A 2008-99158discloses a configuration that produces a wide dynamic range image bycombining a plurality of images with different amounts of exposure. Theprocessing will be explained with reference to FIG. 1. An image captureelement, in capturing moving images, for example, outputs image data fortwo different exposure times within a video rate (30 to 60 fps). Incapturing still images, too, the image data are generated for twodifferent exposure times and output. FIG. 1 is a figure that explainscharacteristics of images (a long exposure time image, a short exposuretime image) that the image capture element generates and that have twodifferent exposure times. The horizontal axis is time (t), and thevertical axis is an accumulated electrical charge (e) in alight-receiving photo diode (PD) that configures a photoelectricconversion element that corresponds to one pixel of a solid imagecapture element.

For example, in a case where the amount of light that thelight-receiving photo diode (PD) is large, that is, where it correspondsto a bright subject, the accumulated electrical charge increases rapidlyas time elapses, as shown in a high brightness region 11 that is shownin FIG. 1. In contrast, in a case where the amount of light that thelight-receiving photo diode (PD) is small, that is, where it correspondsto a dark subject, the accumulated electrical charge increases slowly astime elapses, as shown in a low brightness region 12 that is shown inFIG. 1.

The time from t0 to t3 is equivalent to an exposure time TL foracquiring the long exposure time image. The line that is shown in thelow brightness region 12 shows that the accumulated electrical charge atthe time t3, even as the long exposure time TL, has not reached thesaturation level (unsaturated point Py), and an accurate gray-scaleexpression can be produced according to the gray level of the pixel thatis set using an electrical signal that is produced based on theaccumulated electrical charge (Sa).

However, the line that is shown in the high brightness region 11 clearlyindicates that the accumulated electrical charge has already reached thesaturation level (saturated point Px) before it reaches the time t3.Therefore, in the high brightness region 11, only a pixel value thatcorresponds to an electrical signal at the saturation level is producedfrom the long exposure time image, resulting in a pixel that is blownout.

Accordingly, in the high brightness region 11, the accumulatedelectrical charge is swept out of the light-receiving photo diode (PD)once before the time t3 is reached, for example, at a time t1 (a chargesweeping starting point P1) that is shown in FIG. 1. The charge sweepingdoes not sweep out the entire accumulated electrical charge in thelight-receiving photo diode (PD), but sweeps it down to an intermediatevoltage hold level that is controlled by the photo diode (PD). After thecharge sweeping processing, the light-receiving photo diode (PD) is onceagain exposed to light for a short time that is defined as an exposuretime TS (from t2 to t3 ). That is, a short time exposure is made for theperiod from a short exposure time starting point P2 to a short exposuretime ending point P3, which are both shown in FIG. 1. An accumulatedelectrical charge (Sb) is produced by the short time exposure, and thegray level of the pixel is set using an electrical signal that isproduced based on the accumulated electrical charge (Sb).

Note that in the setting of the pixel value using the electrical signalthat is based on the accumulated electrical charge (Sa) that is producedby the long time exposure in the low brightness region 12 and using theelectrical signal that is based on the accumulated electrical charge(Sb) that is produced by the short time exposure in the high brightnessregion 11, an estimated accumulated electrical charge is computed for acase in which the exposure times are the same in the two regions, anelectrical signal output value that corresponds to the estimatedaccumulated electrical charge is computed, and the pixel value is setbased on the results of the computations.

Combining the short exposure time image and the long exposure time imagein this manner makes it possible to produce an image that has no blownout highlights and a dynamic range that is wide.

Furthermore, Japanese Patent Application Publication No. JP-A 2000-50151discloses a configuration that captures a plurality of images withdifferent amounts of exposure, in the same manner as described inJapanese Patent Application Publication No. JP-A 2008-99158, and in theperforming of the combining processing, inhibits the occurrence of falsecolor that is associated with the combining by comparing the pluralityof the images with the different amounts of exposure, specifying a pixelregion where there is movement, and performing correction.

However, with the configurations that are described in Japanese PatentApplication Publication No. JP-A 2008-99158 and Japanese PatentApplication Publication No. JP-A 2000-50151, it is necessary to capturethe long exposure time image and the short exposure time imageindividually and perform the processing that combines them.

In this manner, using a plurality of images for which the exposure timeshave been changed makes it possible to produce the wide dynamic rangeimage (the high dynamic range (HDR) image), but the problems that aredescribed below, for example, occur in the processing that is based onthe plurality of the images.

First problem: The image capture must be performed a plurality of times.

Second problem: A plurality of images that have been captured atdifferent times are combined, and captured image data with long exposuretimes are used, so the process is vulnerable to the effects of camerainstability.

Known technologies for solving these problems include the knowntechnologies described below, for example.

Technique for Solving First Problem

The pixels within a single solid image capture element are set in twotypes of exposure patterns, that is, pixels with two different types ofexposure time control, pixels that are exposed for a short time, andpixels that are exposed for a long time. This processing makes itpossible to capture images with the short time exposure and the longtime exposure pixels almost simultaneously.

For example, this sort of configuration is disclosed in Japanese PatentApplication Publication No. JP-A 2006-311240 and Japanese PatentApplication Publication No. JP-A 2006-253876.

However, with these configurations, there is a problem in that theexposure times vary from one pixel to the next, so the long exposuretime pixels are more susceptible to blurring than the short exposuretime pixels, and it is difficult to completely eliminate the effects ofcamera instability.

Technique for Solving Second Problem

For example, “SIGGRAPH 2007: Image Deblurring with Blurred/Noisy ImagePairs” proposes a technique for producing a high-quality image from ashort exposure time image with a lot of noise and a long exposure timeimage that has blurring. However, even with the disclosed technique, itis necessary to perform image capture twice, once with a short exposuretime and once with a long exposure time.

Japanese Patent Application Publication No. JP-A 2010-109948 proposes atechnique that, by estimating an amount of movement based on a pluralityof images, corrects blurring in a captured image that has been capturedwith a wide dynamic range.

Furthermore, in “Coded Rolling Shutter Photography: Flexible Space-TimeSampling (ICCP2010),” a technique is disclosed that simultaneouslyperforms image stabilization and processing that combines wide dynamicrange images (HDR combining), based on image data that have beencaptured by varying the exposure times within a single solid imagecapture element one line at a time.

However, in order for the amount of blurring to be estimated when theexposure times are varied one line at a time, the image must be capturedby combining pixels with at least three different exposure times, givingrise to a problem of blurring in the vertical direction. Moreover, thetechnique here described involves processing of an image afterdemosaicing, in which the RGB values have been set for each of the pixelpositions, so a color array such as a Bayer array or the like is nottaken into consideration.

Japanese Patent Application Publication No. JP-A 2010-62785 proposes atechnique that, by implementing a control method for the solid imagecapture element, intermittently captures images with short exposuretimes while a long exposure time image is being captured. However, thereis a problem in that the short exposure time image capture must beperformed a plurality of times, which makes the control morecomplicated.

SUMMARY OF THE INVENTION

In light of the problems that are described above, for example, thepresent disclosure provides an image processing device, an imageprocessing method, and a program that generate an image with a dynamicrange that is wide by using a single captured image, withoutindividually capturing a plurality of images with different exposuretimes.

The present disclosure also provides an image processing device, animage processing method, and a program that generate an image with adynamic range that is wide and in which blurring of the captured imagethat is due to instability is particularly suppressed.

According to an embodiment of the present disclosure, there is providedan image processing device which includes an image capture element thatoutputs long exposure time pixel information and short exposure timepixel information based on image capture processing under a control thatprovides different exposure times for individual pixels, and an imagecorrection portion that inputs the pixel information that the imagecapture element has output and generates a corrected image by performingimage stabilization and dynamic range expansion processing. The imagecorrection portion generates a long exposure time image in which thepixel values have been set on the assumption that all of the pixels havebeen exposed for a long time and a short exposure time image in whichthe pixel values have been set on the assumption that all of the pixelshave been exposed for a short time, computes a point spread function (aPSF) that corresponds to the long exposure time image as a long exposuretime image PSF, and generates the corrected image by using the shortexposure time image, the long exposure time image, and the long exposuretime image PSF.

Further, according to an embodiment of the image processing device ofthe present disclosure, the image correction portion computes a PSF thatcorresponds to the short exposure time image as a short exposure timeimage PSF, and generates the corrected image by using the short exposuretime image, the long exposure time image, the long exposure time imagePSF, and the short exposure time image PSF.

Further, according to an embodiment of the image processing device ofthe present disclosure, the image correction portion generates, based onthe short exposure time image, a first estimated image for which it isassumed that exposure was performed for a long time, computes a firstcorrection amount that makes a difference between the first estimatedimage and the long exposure time image smaller, and generates thecorrected image by performing processing that adds the computed firstcorrection amount to an initial image that has been generated based onthe short exposure time image.

Further, according to an embodiment of the image processing device ofthe present disclosure, the image correction portion performs divisionprocessing that takes a result of a discrete Fourier transform that isbased on a pixel value for a specific color that has been selected fromthe long exposure time image and divides it by a result of a discreteFourier transform that is based on a pixel value for a specific colorthat has been selected from the short exposure time image, and computesthe long exposure time image PSF by performing an inverse discreteFourier transform on the result of the division processing andperforming processing on the result of the inverse discrete Fouriertransform that performs noise removal and identifies a linking componentthat passes through an origin point.

Further, according to an embodiment of the image processing device ofthe present disclosure, the image correction portion performs divisionprocessing that takes a result of a discrete Fourier transform that isbased on a pixel value for a specific color that has been selected fromthe long exposure time image and divides it by a result of a discreteFourier transform that is based on a pixel value for a specific colorthat has been selected from the short exposure time image, and computesthe long exposure time image PSF by performing an inverse discreteFourier transform on the result of the division processing, performingprocessing on the result of the inverse discrete Fourier transform thatperforms noise removal and identifies a linking closed region thatpasses through an origin point, extending the length of a line segmentthat links the origin point and the center of gravity of the closedregion to twice its original length, with the origin point at thecenter, and defining the extended line segment as the long exposure timeimage PSF.

Further, according to an embodiment of the image processing device ofthe present disclosure, the image correction portion computes a PSF thatcorresponds to the short exposure time image as a short exposure timeimage PSF, computing the short exposure time image PSF by multiplying,by a factor that is equal to two times the ratio of an exposure time fora second exposure condition to an exposure time for a first exposurecondition, the length of a line segment that links an origin point andthe center of gravity of the long exposure time image PSF, with theorigin point at the center.

Further, according to an embodiment of the image processing device ofthe present disclosure, the image correction portion includes a firstcorrection amount computation portion that, from a long exposure timeimage PSF that is computed based on the long exposure time image, froman initial image that is generated based on the short exposure timeimage, and from the long exposure time image, computes a firstcorrection amount for the initial image, a second correction amountcomputation portion that, from a short exposure time image PSF that iscomputed based on the short exposure time image, and from the initialimage, computes a second correction amount for the initial image, and anaddition portion that adds the first correction amount and the secondcorrection amount for the initial image.

Further, according to an embodiment of the image processing device ofthe present disclosure, the first correction amount computation portionincludes a first estimated image computation portion that computes afirst estimated image that is a result of estimating an image that issimilar to the long exposure time image, based on the initial image andthe long exposure time image PSF, a subtraction portion that computes afirst difference image that is the difference between the long exposuretime image and the first estimated image, and a first correction amountestimation portion that computes the first correction amount based onthe first difference image and the long exposure time image PSF.

Further, according to an embodiment of the image processing device ofthe present disclosure, the first estimated image computation portionincludes a color-specific PSF computation portion that computes a firstcolor-specific PSF that combines the characteristics of the longexposure time image PSF for each color (phase) of a pixel of the imagecapture element, a first convolution computation portion that performs aconvolution computation for each color of an object pixel of the initialimage using the first color-specific PSF, and a saturation processingportion that takes a results image that has been output from the firstconvolution computation portion and outputs an image by replacing pixelvalues that are not less than a value that is equivalent to a saturatedpixel value of the image capture element with the value that isequivalent to the saturated pixel value.

Further, according to an embodiment of the image processing device ofthe present disclosure, the first correction amount estimation portionincludes an inverse color-specific PSF computation portion that computesa first inverse color-specific PSF in which the long exposure time imagePSF has been inverted symmetrically with respect to a point, a secondconvolution computation portion that performs a convolution computationfor each color of an object pixel of the first difference image usingthe first inverse color-specific PSF, and a multiplication portion thatperforms a correction amount adjustment by multiplying a correctionamount adjustment parameter times a results image that has been outputfrom the second convolution computation portion.

Further, according to an embodiment of the image processing device ofthe present disclosure, the image correction portion includes a demosaicprocessing portion that performs demosaic processing on the longexposure time image and the short exposure time image, and the imagecorrection portion generates the corrected image based on a demosaicedimage that is a processing result of the demosaic processing portion.

Further, according to another embodiment of the present disclosure,there is provided an image processing method that is implemented in animage processing device, including outputting by an image captureelement of long exposure time pixel information and short exposure timepixel information based on image capture processing under a control thatprovides different exposure times for individual pixels, and generatingby an image correction portion of a corrected image by the inputting ofthe pixel information that the image capture element has output and theperforming of image stabilization and dynamic range expansion processingon the pixel information. The generating of the corrected image includesgenerating of a long exposure time image in which the pixel values havebeen set on the assumption that all of the pixels have been exposed fora long time and generating of a short exposure time image in which thepixel values have been set on the assumption that all of the pixels havebeen exposed for a short time, computing of a point spread function (aPSF) that corresponds to the long exposure time image as a long exposuretime image PSF, and generating of the corrected image by using the shortexposure time image, the long exposure time image, and the long exposuretime image PSF.

Further, according to another embodiment of the present disclosure,there is provided a program that causes image processing to be performedin an image processing device, including outputting by an image captureelement of long exposure time pixel information and short exposure timepixel information based on image capture processing under a control thatprovides different exposure times for individual pixels, and generatingby an image correction portion of a corrected image by the inputting ofthe pixel information that the image capture element has output and theperforming of image stabilization and dynamic range expansion processingon the pixel information. The generating of the corrected image includesgenerating of a long exposure time image in which the pixel values havebeen set on the assumption that all of the pixels have been exposed fora long time and generating of a short exposure time image in which thepixel values have been set on the assumption that all of the pixels havebeen exposed for a short time, computing of a point spread function (aPSF) that corresponds to the long exposure time image as a long exposuretime image PSF, and generating of the corrected image by using the shortexposure time image, the long exposure time image, and the long exposuretime image PSF.

Note that the program according to the present disclosure is a programthat can be provided in a storage medium or communication medium that isprovided in a computer-readable form for an information processingdevice or a computer system that is capable of executing various typesof program code, for example. Providing this sort of program in acomputer-readable form makes it possible to implement the processingaccording to the program in the information processing device or thecomputer system.

The object, features, and advantages of the present disclosure will bemade clear later by a more detailed explanation that is based on theembodiments of the present disclosure and the appended drawings.

According to the example of the present disclosure, a device and amethod are achieved that generate an image with reduced blurring and awide dynamic range based on a single captured image. Specifically, thedevice includes an image capture element that outputs long exposure timepixel information and short exposure time pixel information based onimage capture processing under a control that provides differentexposure times for individual pixels, and also includes an imagecorrection portion that inputs the pixel information that the imagecapture element has output and generates a corrected image by performingimage stabilization and dynamic range expansion processing.

The image correction portion generates a long exposure time image inwhich the pixel values have been set on the assumption that all of thepixels have been exposed for a long time and a short exposure time imagein which the pixel values have been set on the assumption that all ofthe pixels have been exposed for a short time, computes a point spreadfunction (a PSF) that corresponds to the long exposure time image as along exposure time image PSF, and generates the corrected image by usingthe short exposure time image, the long exposure time image, and thelong exposure time image PSF. The corrected image is generated as a widedynamic range image that utilizes the pixel information for the longexposure time image and the pixel information for the short exposuretime image. Utilizing the pixel information for the short exposure timeimage, in which there is little blurring, also makes the corrected imagea high quality corrected image in which the blurring is suppressed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a figure that explains processing that produces an image witha wide dynamic range by combining a plurality of images with differentamounts of exposure;

FIG. 2 is a figure that shows an example of a configuration of an imageprocessing device according to the present disclosure;

FIG. 3 is a figure that explains an example of a configuration of andprocessing by an image capture element in the image processing deviceaccording to the present disclosure;

FIG. 4 is a figure that explains an example of a configuration of animage correction portion in the image processing device according to thepresent disclosure;

FIG. 5 is a figure that explains an example of processing by aninterpolation processing portion of the image correction portion in theimage processing device according to the present disclosure;

FIG. 6 is a figure that explains an example of a configuration of andprocessing by a PSF estimation portion of the image correction portionin the image processing device according to the present disclosure;

FIG. 7 is a figure that explains an example of the configuration of andthe processing by the PSF estimation portion of the image correctionportion in the image processing device according to the presentdisclosure;

FIG. 8 is a figure that explains an example of a configuration of andprocessing by an image stabilization and dynamic range expansionprocessing portion of the image correction portion in the imageprocessing device according to the present disclosure;

FIG. 9 is a figure that explains another example of the configuration ofand the processing by the image stabilization and dynamic rangeexpansion processing portion of the image correction portion in theimage processing device according to the present disclosure;

FIG. 10 is a figure that explains an example of a configuration of andprocessing by a first correction amount computation portion of the imagestabilization and dynamic range expansion processing portion of theimage correction portion in the image processing device according to thepresent disclosure;

FIG. 11 is a figure that explains an example of a configuration of andprocessing by a color-specific PSF and inverse PSF color-specificgeneration portion of the image stabilization and dynamic rangeexpansion processing portion of the image correction portion in theimage processing device according to the present disclosure;

FIG. 12 is a figure that explains another example of the configurationof and the processing by the image stabilization and dynamic rangeexpansion processing portion of the image correction portion in theimage processing device according to the present disclosure;

FIG. 13 is a figure that explains another example of the configurationof the image capture element in the image processing device according tothe present disclosure; and

FIG. 14 is a figure that explains an example of exposure control in theimage processing device according to the present disclosure.

DETAILED DESCRIPTION OF EMBODIMENT

Hereinafter, preferred embodiments of the present disclosure will bedescribed in detail with reference to the appended drawings. Note that,in this specification and the appended drawings, structural elementsthat have substantially the same function and structure are denoted withthe same reference numerals, and repeated explanation of thesestructural elements is omitted.

Hereinafter, an image processing device, an image processing method, anda program according to the present disclosure will be explained withreference to the drawings. The explanation will cover the items below inorder.

1. Example of an overall configuration of the image processing device

2. Example of a configuration of an image capture element

3. Details of a configuration of and processing by an image correctionportion

(3-1) Details of a configuration of and processing that is performed byan interpolation processing portion

(3-2) Details of a configuration of and processing that is performed bya PSF estimation portion

(3-3) Details of a configuration of and processing that is performed byan image stabilization and dynamic range expansion processing portion

4. Examples of other configurations

5. Examples of other processing and concomitant configurations and theireffects

6. Summary of the configurations of the present disclosure

1. Example of an Overall Configuration of the Image Processing Device

First, an example of an overall configuration of the image processingdevice according to the present disclosure will be explained withreference to FIG. 2.

FIG. 2 is a block diagram that shows a configuration of an image capturedevice that is an example of the image processing device according tothe present disclosure. Light that enters through an optical lens 101 isincident upon an image capture element 102 that is configured from aCMOS image sensor or the like, for example, where the light undergoesphotoelectric conversion and image data are output. The output imagedata are input to an image correction portion 103.

Note that the configuration of the image capture element 102 is suchthat one of the exposure time and the read time is cyclicallycontrolled, one of one line at a time and one pixel at a time, under thecontrol of a control portion 105, such that long exposure time pixelsand short exposure time pixels are defined.

The image correction portion 103 performs image correction processingthat includes dynamic range expansion processing and image stabilizationprocessing. The details of this processing will be explained in detailat a later stage. A corrected image that has been generated by the imagecorrection portion 103 is input to a signal processing portion 104. Thesignal processing portion 104 performs signal processing that isgenerally performed in a camera, such as white balance (WB) adjustment,gamma correction, and the like, then generates an output image 120. Theoutput image 120 is stored in a storage portion that is not shown in thedrawings. Alternatively, the output image 120 is output to a displayportion.

The control portion 105 outputs control signals to various portions andperforms various types of processing control in accordance with aprogram that is stored in a memory, for example, that is not shown inthe drawings.

2. Example of a Configuration of the Image Capture Element 102

One of the exposure time and the read time of the image capture element102, which is configured from a CMOS image sensor or the like, forexample, is cyclically controlled, one of one line at a time and onepixel at a time, by the control portion 105.

Specifically, as shown in part (a) of FIG. 3, for example, long exposuretime pixels and short exposure time pixels are defined.

Note that in the present example, the pixel array of the image captureelement 102 is defined as a Bayer array. However, this is merely oneexample, and the processing according to the present disclosure can alsobe applied to other pixel arrays.

Part (b) of FIG. 3 shows an example of the settings for the exposurestart times and the exposure end times for the long exposure time pixelsand the short exposure time pixels.

In the present example, the ratio of the exposure times for the longexposure time pixels and the short exposure time pixels is 4:1, and theexposure start times are set to different times, while the exposure endtimes (the read times) are set to be the same.

Note that these settings are merely examples, and various other settingcan also be made.

However, the image capture element 102 has a configuration that iscapable of acquiring pixel information from pixels for which a pluralityof different exposure times have been set for a single round of imagecapture.

3. Details of a Configuration of and Processing by the Image CorrectionPortion 103

Next, Details of a configuration and processing of the image correctionportion 103 will be explained with reference to FIG. 4 and subsequentdrawings.

As was explained above with reference to FIG. 2, the output of the imagecapture element 102, which contains the pixels for which the pluralityof the different exposure times have been set, is input to the imagecorrection portion 103, where the corrected image, for which the dynamicrange has been expanded and image stabilization has been performed, isgenerated and output to the signal processing portion 104.

As shown in FIG. 4, the image correction portion 103 includes aninterpolation processing portion 201, a PSF estimation portion 202, andan image stabilization and dynamic range expansion processing portion203. Hereinafter, details of the processing that is performed by thesestructural portions will be explained in the order shown below.

(3-1) Details of a configuration of and processing that is performed bythe interpolation processing portion 201

(3-2) Details of a configuration of and processing that is performed bythe PSF estimation portion 202

(3-3) Details of a configuration of and processing that is performed bythe image stabilization and dynamic range expansion processing portion203

(3-1) Details of a Configuration of and Processing that is Performed bythe Interpolation Processing Portion 201

First, the details of the configuration of and processing that isperformed by the interpolation processing portion 201 that is shown inFIG. 4 will be explained.

As shown in FIG. 4, the output of the image capture element 102 is inputto the interpolation processing portion 201 of the image correctionportion 103.

The processing in the interpolation processing portion 201 will beexplained with reference to FIG. 5.

The interpolation processing portion 201 inputs from the image captureelement 102 an output image 301 in which different exposure times havebeen set one pixel at a time, and the interpolation processing portion201 generates two exposure time images, a first exposure time image (along exposure time image) 321 and a second exposure time image (a shortexposure time image) 322.

The interpolation processing portion 201 performs interpolationprocessing such that various exposure patterns are formed for all of thepixels. Various interpolation methods can be applied in theinterpolation processing, such as a method of interpolating by filters,such as linear interpolation or the like, a method of detecting thedirections of edges in the image and interpolating based on those edges,and the like.

For example, the first exposure time image (a long exposure time image)321 uses the long exposure time pixels that are contained in the outputimage 301 from the image capture element 102 in their existing form.When the long exposure time pixels are interpolated among the positionsof the short exposure time pixels in the image capture element 102,processing can be performed that computes the interpolated pixel valuesby the aforementioned method of interpolating by filters, such as linearinterpolation or the like, based on the pixel values of the surroundinglong exposure time pixels. The interpolated pixel values can also becomputed by detecting the directions of the edges in the image, thentaking the directions of the edges into consideration by assigninggreater weightings to pixel values in the directions in which thedifferences in the pixel values are low.

The second exposure time image (a short exposure time image) 322 usesthe short exposure time pixels that are contained in the output image301 from the image capture element 102 in their existing form. When theshort exposure time pixels are interpolated among the positions of thelong exposure time pixels in the image capture element 102, processingcan be performed that computes the interpolated pixel values by theaforementioned method of interpolating by filters, such as linearinterpolation or the like, based on the pixel values of the surroundingshort exposure time pixels. The interpolated pixel values can also becomputed by detecting the directions of the edges in the image, thentaking the directions of the edges into consideration by assigninggreater weightings to pixel values in the directions in which thedifferences in the pixel values are low.

By performing the interpolation processing on the output image 301 fromthe image capture element 102, as shown in FIG. 5, the interpolationprocessing portion 201 generates the first exposure time image (the longexposure time image) 321 with a long exposure time Bayer array and thesecond exposure time image (the short exposure time image) 322 with ashort exposure time Bayer array

As shown in FIG. 4, the images 321, 322 that the interpolationprocessing portion 201 has generated with the two different exposuretimes are input to the PSF estimation portion 202 and the imagestabilization and dynamic range expansion processing portion 203.

(3-2) Details of a Configuration of and Processing that is Performed bythe PSF Estimation Portion 202

Next, the details of the configuration of and processing that isperformed by the PSF estimation portion 202 that is shown in FIG. 4 willbe explained.

The PSF estimation portion 202 performs processing that estimates apoint spread function (PSF).

The processing that the PSF estimation portion 202 performs will beexplained with reference to FIGS. 6 and 7.

FIG. 6 shows a first half of the PSF estimation portion 202, and FIG. 7shows a second half of the PSF estimation portion 202.

First, the processing in the first half of the PSF estimation portion202 will be explained with reference to FIG. 6.

As shown in FIG. 6, the PSF estimation portion 202 inputs the two imagesthat have been generated by the interpolation processing portion 201,that is, the first exposure time image (the long exposure time image)321 and the second exposure time image (the short exposure time image)322.

A first pixel selection portion 351 extracts only G pixels from thefirst exposure time image (the long exposure time image) 321.

A second pixel selection portion 361 extracts only G pixels from thesecond exposure time image (the short exposure time image) 322.

In the processing that extracts the G pixels, the pixel values for thepixels other than the G pixels are set to zero.

An exposure correction portion 362, by multiplying the G pixel data thathave been extracted from the second exposure time image (the shortexposure time image) 322 times an exposure ratio, generates an imagethat has almost the same brightness as the first exposure time image321. This will be explained in detail at a later stage.

Note that under ordinary circumstances, it is necessary to interpolatethe G pixels among all of the pixel positions by using bilinearinterpolation or the like, but this processing is equivalent to applyinga low pass filter after a Fourier transform, so in the present example,the cost of computation is reduced by instead eliminating the highfrequency components after a discrete Fourier transform.

The data that include only the pixel values for the G pixels are outputto first and second discrete Fourier transform portions 352, 363.

Note that, as will be described later, the G pixels from the secondexposure time image 322, which is the short exposure time image, areoutput to the second discrete Fourier transform portion 363 after theexposure correction in the exposure correction portion 362.

The first and second discrete Fourier transform portions 352, 363 eachperform a discrete Fourier transform using a window function.

The first discrete Fourier transform portion 352 computes a longexposure time G pixel Fourier transform result based on the G pixel datathat have been generated from the first exposure time image (the longexposure time image) 321.

The second discrete Fourier transform portion 363 computes a shortexposure time G pixel Fourier transform result based on the G pixel datathat have been generated by performing the exposure correction on thesecond exposure time image (the short exposure time image) 322.

The Fourier transform results are output to a PSF estimation computationportion 371.

The PSF estimation computation portion 371 computes the quotient of thetwo Fourier transform results. That is, the quotient is computed asshown below.Quotient=(long exposure time G pixel Fourier transform result)/(shortexposure time G pixel Fourier transform result)

The quotient is output to a low pass filter portion 372.

The low pass filter portion 372 performs processing that sets to zerothe high frequency components that are included in the quotient. Theresult of eliminating the high frequencies is output to an inversediscrete Fourier transform portion 373.

The inverse discrete Fourier transform portion 373 computes an inversediscrete Fourier transform.

The inverse discrete Fourier transform result that has been generated bythe inverse discrete Fourier transform portion 373 is output to a noiseremoval and closed region selection portion 374.

The noise removal and closed region selection portion 374 removes noisefrom the inverse discrete Fourier transform result that has beengenerated by the inverse discrete Fourier transform portion 373, selectsa closed region, performs PSF estimation, and generates a first PSF (along exposure time image PSF) 381.

A specific example of the processing in the PSF estimation portion 202will be explained.

Assume that the positions of the pixels that make up an image areexpressed as coordinate positions (x, y), that f(x, y) defines the pixelvalues at the individual pixel positions (x, y) in the second exposuretime image (the short exposure time image) 322, that g(x, y) defines thepixel values at the individual pixel positions (x, y) in the firstexposure time image (the long exposure time image) 321, and that p(x, y)defines the values of the PSF (the point spread function) at theindividual pixel positions (x, y).

However, for purposes of this explanation, f(x, y) and g(x, y) areassumed to be monochromatic images instead of Bayer arrays, andsaturation (blowing out) of the pixels is ignored. The first exposuretime image (the long exposure time image) 321 is affected by the camerainstability, so it is expressed as shown below.

Note that the asterisk (*) indicates a convolution computation.g(x,y)=p(x,y)*f(x,y)  Equation 1

In other words, the pixel values g(x, y) at the individual pixelpositions (x, y) in the first exposure time image (the long exposuretime image) 321 are computed by performing a convolution computation ofthe PSF and the second exposure time image (the short exposure timeimage) 322.

The results of the discrete Fourier transforms for f(x, y) for thesecond exposure time image (the short exposure time image) 322, g(x, y)for the first exposure time image (the long exposure time image) 321,and p(x, y) for the PSF are expressed as described below.

The discrete Fourier transform result for f(x, y) for the secondexposure time image (the short exposure time image) 322 is F(u, v). Thediscrete Fourier transform result for g(x, y) for the first exposuretime image (the long exposure time image) 321 is G(u, v). The discreteFourier transform result for p(x, y) for the PSF is P(u, v).

When the discrete Fourier transform results are used, Equation 1 abovecan be expressed as Equation 2 below.G(u,v)=P(u,v)·F(u,v)  Equation 2

Note that “·” means the multiplication of each frequency component.

In other words, the discrete Fourier transform result G(u, v) for g(x,y) for the first exposure time image (the long exposure time image) 321is computed by multiplying the discrete Fourier transform result P(u, v)for p(x, y) for the PSF times the discrete Fourier transform result F(u,v) for f(x, y) for the second exposure time image (the short exposuretime image) 322, one frequency component at a time.

Based on Equation 1 and Equation 2, Equation 3 can be used to computep(x, y) for the PSF.P(u,v)=G(u,v)/F(u,v)  Equation 3

The discrete Fourier transform result P(u, v) for p(x, y) is computedaccording to Equation 3, and the inverse discrete Fourier transform ofthe computed discrete Fourier transform result P(u, v) may also becomputed.

Note that “/” in Equation 3 above indicates the division of eachfrequency component.

This procedure makes it possible to derive f(x, y) for the secondexposure time image (the short exposure time image) 322 and g(x, y) forthe first exposure time image (the long exposure time image) 321, andp(x, y) for the PSF (the point spread function) can also be derivedusing these results.

However, the image brightnesses of a short exposure time image and along exposure time image are actually different, so a situation isconceivable in which an area that can be seen in a short exposure timeimage is completely saturated and blown out in a long exposure timeimage. In that case, Equation 1 above is invalidated by the effects ofthe saturation.

Accordingly, processing is actually performed in the exposure correctionportion 362 that multiplies the exposure ratio (4 in the example in thepresent disclosure) times the G pixel data that have been generated fromthe second exposure time image (the short exposure time image) 322, thenreplaces the pixel values that are greater than a saturated pixel valuewith the saturated pixel value. In the G pixel data that are produced bythis processing and in the G pixel data that have been generated fromthe first exposure time image (the long exposure time image) 321, thepositions of the saturated pixels match almost perfectly, so it ispossible to eliminate the effects of the saturation of the pixels.

Furthermore, the effects of noise are significant only in processingthat divides by the frequency space, as in the technique that is shownby Equation 3 above, and in a case where the discrete Fourier transformresult F(u, v) for f(x, y) for the second exposure time image (the shortexposure time image) 322 is close to zero, Equation 3 becomes a case ofdivision by zero. That is, as shown in FIG. 6, the result that iscomputed by Equation 3 above in the inverse discrete Fourier transformportion 373 becomes a PSF that includes a large amount of noise.

Accordingly, in the image capture device according to the presentdisclosure, the processing is performed by taking into consideration theproperties of the image capture device that pixels that are capturedunder a first exposure condition and pixels that are captured under asecond exposure condition are captured almost simultaneously and thatthe PSF where there is camera instability is in a coupled form, andsmall amounts of high frequency noise are not included.

As is understood from the exposure times that have been explained withreference to part (b) of FIG. 3, the first exposure time image (the longexposure time image) 321 is being exposed during the exposure period forthe second exposure time image (the short exposure time image) 322. Thismeans that the PSF that is generated based on the two images that havethe overlapping exposure periods will definitely pass through an originpoint.

The noise removal and closed region selection portion 374 that is shownin FIG. 6 takes advantage of this when it inputs the result that hasbeen computed by Equation 3 above in the inverse discrete Fouriertransform portion 373, removes the high frequency noise from the input,and selects a closed region that passes through the origin point. Anormalization portion 375 performs normalization processing such thatthe sum of the pixel values in the selected closed region will equal theexposure ratio (4.0 in the example in the present disclosure). The firstPSF (the long exposure time image PSF) 381 is generated with littlenoise as a result.

Next, the second half of the PSF estimation portion 202 will beexplained with reference to FIG. 7.

In the second half of the PSF estimation portion 202, a higher precisionfirst PSF (the long exposure time image PSF) and a second PSF (a shortexposure time image PSF) are computed by assuming that the PSF has alinear form.

The first PSF (the long exposure time image PSF) 381 that was generatedin the first half of the PSF estimation portion 202 that was explainedwith reference to FIG. 6 is input, and the center of gravity of the PSFis detected by a center of gravity detection portion 401. Because theform of the PSF is assumed to be linear, under normal circumstances, aline segment that links the origin point and the center of gravitybecomes a line segment that is extended to twice its original length,with the origin point at the center. Accordingly, in a PSF correctionprocessing portion 402, a first high precision PSF 421 is generated byperforming computation that generates the line segment that has beenextended to twice its original length, with the origin point at thecenter.

The length of a second PSF (a short exposure time image PSF) 422 becomesdata that are computed by multiplying the exposure ratio times thelength of the first PSF (the long exposure time image PSF) 381.Accordingly, in a second PSF computation portion 403, an extended linesegment with the origin point at the center is generated by multiplyingthe line segment that links the origin point and the center of gravityof the first PSF (the long exposure time image PSF) 381 by the result ofthe equation below.2×(exposure time for second exposure condition)÷(exposure time for firstexposure condition)

The result is output as the second PSF (the short exposure time imagePSF) 422.

(3-3) Details of the Configuration of and the Processing that isPerformed by the Image Stabilization and Dynamic Range ExpansionProcessing Portion 203

Next, the details of the configuration of and processing that isperformed by the image stabilization and dynamic range expansionprocessing portion 203 that is shown in FIG. 4 will be explained.

The image stabilization and dynamic range expansion processing portion203 sets an initial image based on the second exposure time image (theshort exposure time image) 322, which has little blurring, computes acorrection amount such that there will be little difference between anactually observed image and an estimated image in which the initialimage has been blurred by the PSF, and updates the initial image.

This processing is able to expand the dynamic range while removing noiseand remedying the deterioration of the gray levels in dark areas of theinitial image, as well as eliminating the effects of camera instability.

A specific example of the processing will be explained.

When the initial image is updated using a first correction amount thathas been computed based on the first exposure time image (the longexposure time image) 321, the computation below is performed.h′(x,y)=h(x,y)+λp′(x,y)*{g(x,y)−p(x,y)*h(x,y)}  Equation 4

Note that in Equation 4 above, p′(x, y) is an inverse PSF that isderived by inverting the PSF symmetrically with respect to a point, λ isa parameter for adjustment (a parameter that is set in advance or thatis set by a user), h(x, y) is the initial image, h′(x, y) is the updatedinitial image that is the result of the correction, and * indicates aconvolution computation.

Equation 4 above is expressed without distinguishing among colors, butactually, convolution computation that will be described later isperformed by using a color-specific PSF and an inverse color-specificPSF for each color (phase) of the object pixel.

A detailed configuration of the image stabilization and dynamic rangeexpansion processing portion 203, and the processing that it performs,will be explained with reference to FIG. 8.

From the interpolation processing portion 201, which has been explainedwith reference to FIGS. 4 and 5, the image stabilization and dynamicrange expansion processing portion 203 inputs the two images, the firstexposure time image (the long exposure time image) 321 and the secondexposure time image (the short exposure time image) 322, that have beengenerated by the interpolation processing portion 201.

First, for the images 321, 322 that have been input, the imagestabilization and dynamic range expansion processing portion 203performs bit shifting of the configuring pixel values in first andsecond bit shift portions 511 and 521, respectively. The bit shiftingproduces a first high gradation image 512 and a second high gradationimage 522 that have high bit gray levels.

For example, in a case where the pixel values in the input images, thatis, the first exposure time image (the long exposure time image) 321 andthe second exposure time image (the short exposure time image) 322, arein the form of 10-bit data, the first and second bit shift portions 511and 521 set the individual pixel values in the images as 16-bit data.

An initial image 551 that is shown in FIG. 8 initially uses an image onwhich noise removal has been performed by a noise removal portion 531,in contrast to the second high gradation image 522 that has beengenerated based on the second exposure time image (the short exposuretime image) 322.

Next, in a first correction amount computation portion 532, a firstcorrection amount is computed based on the first high gradation image512 that has been generated based on the first exposure time image (thelong exposure time image) 321, the first PSF (the long exposure timeimage PSF) 381 that has been generated by the PSF estimation portion202, and the initial image 551.

The correction amount computation processing that is performed by thefirst correction amount computation portion 532 will be described ingreater detail at a later stage.

Note that the first high precision PSF 421 that was explained withreference to FIG. 7 may also be used instead of the first PSF (the longexposure time image PSF) 381.

In a second correction amount computation portion 533, a secondcorrection amount is also computed based on the second high gradationimage 522 that has been generated based on the second exposure timeimage (the short exposure time image) 322, the second PSF (the shortexposure time image PSF) 422 that has been generated by the PSFestimation portion 202, and the initial image 551.

Next, in an addition portion 534, the first correction amount and thesecond correction amount are added to the initial image 551 to produce acorrected output image 552.

Note that the corrected output image 552 may also be set as the initialimage 551, the same processing may be repeated, and the corrected outputimage 552 may be successively updated to ultimately generate a finalcorrected output image 552.

Note that, as stated previously, in the processing in the firstcorrection amount computation portion 532 that computes the firstcorrection amount that corresponds to the first exposure time image (thelong exposure time image) 321, either one of the first PSF (the longexposure time image PSF) 381 that was generated in the PSF estimationportion 202 and that was explained with reference to FIG. 6 and thefirst high precision PSF 421 that was generated in the PSF estimationportion 202 and that was explained with reference to FIG. 7 can be used.

Furthermore, the noise removal portion 531 is added processing, and theimage stabilization and dynamic range expansion processing portion 203may also be configured such that the noise removal portion 531 isomitted.

Note that in a case where it is determined that the exposure time forthe second exposure time image 322, which is the short exposure timeimage, is sufficiently short that there is no blurring in the secondexposure time image 322, it is also possible to configure the imagecapture device such that the computation of the second PSF (the shortexposure time image PSF) 422 in the PSF estimation portion 202, whichwas explained earlier with reference to FIG. 7, and the secondcorrection amount computation portion 533 in the image stabilization anddynamic range expansion processing portion 203 that is shown in FIG. 8are omitted.

In that case, the image stabilization and dynamic range expansionprocessing portion 203 can be configured as shown in FIG. 9, forexample.

Note that the noise removal portion has been omitted in FIG. 9.

The corrected output image 552 that is produced as the result of theprocessing in the image stabilization and dynamic range expansionprocessing portion 203 that is shown in FIG. 8 is output to the signalprocessing portion 104 that is shown in FIG. 2.

By performing gray-level correction, demosaicing, gamma correction, andthe like as necessary, the signal processing portion 104 generates andoutputs the final output image 120.

Details of the First Correction Amount Computation Portion 532

Next, the details of the first correction amount computation portion 532that is shown in FIG. 8 will be explained with reference to FIGS. 10 and11.

Note that the second correction amount computation portion 533 also hasthe same configuration.

First, first color-specific PSFs 711 and first inverse color-specificPSFs 712 are generated in a color-specific PSF and inversecolor-specific PSF generation portion 701 in accordance with the colorarray in the image capture element 102.

Note that “color-specific” (phase-specific) means that, in the case of aBayer array, the PSFs are generated to correspond to the three RGBcolors, such that a PSF and an inverse PSF that correspond to R, a PSFand an inverse PSF that correspond to G, and a PSF and an inverse PSFthat correspond to B are generated.

The first color-specific PSFs 711 are input to a first convolutioncomputation portion 702.

The first convolution computation portion 702 performs a convolutioncomputation using the first color-specific PSFs 711 that correspond tothe colors of the object pixel in the initial image 551 (refer to FIG.8) that was generated based on the second exposure time image (the shortexposure time image) 322.

The convolution computation results that are generated by the firstconvolution computation portion 702 are output to a saturationprocessing portion 703.

The saturation processing portion 703 inputs the convolution computationresults that have been generated by the first convolution computationportion 702 and performs saturation processing that takes the pixelvalues that are not less than a predetermined threshold value andsaturates them at the threshold value.

This processing generates an image (a first estimated image 721) that itcan be predicted, based on the initial image 551 and the first PSF (thelong exposure time image PSF) 381, will be produced when an image iscaptured under the first exposure condition (the long exposure time).

The first correction amount computation portion 532 computes the firstcorrection amount such that the difference between the first estimatedimage 721 and the first exposure time image (the long exposure timeimage) 321 that was generated by the interpolation processing portion201 will become smaller.

A subtraction portion 704 generates a first difference image 722 thatdefines the difference between the first estimated image 721 and thefirst exposure time image (the long exposure time image) 321 that wasgenerated by the interpolation processing portion 201.

The subtraction portion 704 produces the first difference image 722 byperforming processing that subtracts the pixel values of the firstestimated image 721 from the corresponding pixel values in the firstexposure time image (the long exposure time image) 321 that wasgenerated by the interpolation processing portion 201.

Next, a second convolution computation portion 705 performs aconvolution computation using the first inverse color-specific PSFs 712for each of the colors (R, G, B) of the object pixels, which areselected sequentially.

Finally, the first correction amount is produced by a multiplicationportion 706 that multiplies the computation result from the secondconvolution computation portion 705 times a correction amount adjustmentparameter 731.

One of a predetermined value and a value that is set by the user can beused for the correction amount adjustment parameter 731.

Next, a detailed configuration of the color-specific PSF and inversecolor-specific PSF generation portion 701 and its processing will beexplained with reference to FIG. 11.

In this processing, color-specific PSFs 821, 822, the properties ofwhich are matched to the colors (the phases) of the solid image captureelement, are computed in order to perform image stabilization of thecolor array (the Bayer array in the present example) of the imagecapture element 102 in its existing form.

At the same time, inverse color-specific PSFs 823, 824 are computed thatinvert the color-specific PSFs symmetrically with respect to a point.

First, the pixels for each color (phase) (R, G, B) in the first PSF (thelong exposure time image PSF) 381 that has been input to thecolor-specific PSF and inverse color-specific PSF generation portion 701are extracted in a first pixel selection portion 802. In the case of theBayer array, for example, the G pixels are extracted in a checkerboardpattern, and the R and B pixels are extracted for every second pixel.

Thereafter, processing is performed in first and second normalizationportions 804 a, 804 b that performs normalization according to theexposure ratio of the initial image 551 and the first exposure timeimage (the long exposure time image) 321.

In the present example, as explained previously with reference to FIG.3, the exposure ratio of the short exposure time to the long exposuretime is 1:4, so the normalization is performed such that the totalvalues for the color-specific PSFs 821, 822 are each 4.0.

Furthermore, in an inversion portion 801, the first PSF (the longexposure time image PSF) 381 is inverted symmetrically with respect to apoint, and in a second pixel selection portion 803, the pixels areextracted that correspond to the individual colors (R, G, B). Then, inthird and fourth normalization portions 804 c, 804 d, inversecolor-specific PSFs 823, 824 are output by performing normalization suchthat the total values are the inverse of the exposure ratio, that is,0.25.

4. Examples of Other Configurations

Next, examples of other configurations that are different from theexample that is described above will be explained.

As explained previously, in a case the exposure time for the secondexposure condition (the short exposure time) is sufficiently short thatthere is no blurring in the captured image, it is possible to configurethe image capture device such that the processing that generates thesecond PSF (the short exposure time image PSF) 422 in the PSF estimationportion 202 that is shown in FIG. 7 and the second correction amountcomputation portion 533 in the image stabilization and dynamic rangeexpansion processing portion 203 that is shown in FIG. 8 are omitted, asshown in FIG. 9.

Furthermore, a technique that combines the Bayer arrays in theirexisting form was shown in FIG. 8 as one example of a configuration, butit is also possible to perform demosaic processing first, and then toperform the processing on an image in which the RGB pixel values thatcorrespond to each of the pixel positions have been set.

An example of the configuration of the image stabilization and dynamicrange expansion processing portion 203 is shown in FIG. 12 in which thedemosaic processing is performed first, after which the imagestabilization and the dynamic range expansion processing are performed.

First, as shown in FIG. 12, the images that have been generated by theinterpolation processing portion 201, that is, the first exposure timeimage (the long exposure time image) 321 and the second exposure timeimage (the short exposure time image) 322, are input to first and seconddemosaic processing portions 901, 902, respectively.

The first and second demosaic processing portions 901, 902 perform thedemosaic processing on the first exposure time image (the long exposuretime image) 321 and the second exposure time image (the short exposuretime image) 322, respectively, converting them into images in which thethree RGB colors have been set for each of the pixel positions.

The subsequent processing basically performs the same sort of processingas the processing that was explained previously with reference to FIG.8, performing it separately for the R, G, and B images. However, thethree RGB colors have been set for all of the pixel positions, so it isnot necessary to use the color-specific PSF and the inversecolor-specific PSF.

Furthermore, utilizing the characteristic that blurring is lessnoticeable in a color difference signal than in a brightness signal, thecomputed amount can be reduced by converting the RGB image into a YCbCrspace and performing the processing only with respect to the Y signal.

An example of the settings for the long exposure time pixels and theshort exposure time pixels in the pixel array of the CMOS image sensor,for example, that serves as the image capture element 102 was explainedearlier with reference to FIG. 3.

The pattern that is shown in FIG. 3 is merely one example, and it ispossible to set various other types of pixel patterns. It is possible toapply the present technique by designing the image capture device suchthat the interpolation processing portion 201 and the color-specific PSFand inverse color-specific PSF generation portion 701 are combined inthe pixel array.

For example, the present technique can be applied to an image sensorwith the pixel arrays that are shown in parts (A) and (B) of FIG. 13, aswell as an image sensor with a color filter that is a non-Bayer array,as shown in part (C) of FIG. 13. Furthermore, in the example of theexposure control that was explained with reference to FIG. part (b) ofFIG. 3, the control was implemented such that the exposure end times forthe two exposure patterns are the same, but different exposure times mayalso be used, and the control may be performed such that the exposurestart times are the same and the exposure end times are different, asshown in FIG. 14, for example.

5. Examples of Other Processing and Concomitant Configurations and theirEffects

As the image processing device (the image capture device) of the presentdisclosure, a technique has been proposed above that performs dynamicrange expansion and image stabilization simultaneously by implementingexposure control for the image capture element (the image sensor) andprocessing for the output of the image capture element. The techniqueaccording to the present disclosure can also be used in combination withknown optical image stabilization.

To summarize the configuration of the present disclosure and itseffects, the effects described below are achieved, for example.

-   -   A stabilized image with an expanded dynamic range can be        produced from a single captured image by signal processing. That        means that an image is produced in which there is little noise,        from dark areas to bright areas, and the effects of blurring are        suppressed.    -   The image processing that is described above can also be applied        to a solid image capture element with a simpler pixel array and        control method than the known control method. Specifically, the        processing can be performed on a pixel array with fewer exposure        conditions (two) than in “Coded Rolling Shutter Photography:        Flexible Space-Time Sampling” (ICCP2010), and the exposure        control can also be configured simply and easily.    -   The color arrays for the input image and the output image can be        processed as the existing Bayer arrays, so the compatibility        with existing camera signal processing is high.

6. Summary of the Configurations of the Present Disclosure

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

Additionally, the present technology may also be configured as below.

-   (1)

An image processing device, including:

an image capture element that outputs long exposure time pixelinformation and short exposure time pixel information based on imagecapture processing under a control that provides different exposuretimes for individual pixels; and

an image correction portion that inputs the pixel information that theimage capture element has output and generates a corrected image byperforming image stabilization and dynamic range expansion processing,

wherein the image correction portion

generates a long exposure time image in which the pixel values have beenset on the assumption that all of the pixels have been exposed for along time and a short exposure time image in which the pixel values havebeen set on the assumption that all of the pixels have been exposed fora short time,

computes a point spread function (a PSF) that corresponds to the longexposure time image as a long exposure time image PSF, and

generates the corrected image by using the short exposure time image,the long exposure time image, and the long exposure time image PSF.

-   (2)

The image processing device according to (1),

wherein the image correction portion

computes a PSF that corresponds to the short exposure time image as ashort exposure time image PSF, and

generates the corrected image by using the short exposure time image,the long exposure time image, the long exposure time image PSF, and theshort exposure time image PSF.

-   (3)

The image processing device according to (1) or (2),

wherein the image correction portion generates, based on the shortexposure time image, a first estimated image for which it is assumedthat exposure was performed for a long time, computes a first correctionamount that makes a difference between the first estimated image and thelong exposure time image smaller, and generates the corrected image byperforming processing that adds the computed first correction amount toan initial image that has been generated based on the short exposuretime image.

-   (4)

The image processing device according to any one of (1) to (3),

wherein the image correction portion

performs division processing that takes a result of a discrete Fouriertransform that is based on a pixel value for a specific color that hasbeen selected from the long exposure time image and divides it by aresult of a discrete Fourier transform that is based on a pixel valuefor a specific color that has been selected from the short exposure timeimage, and

computes the long exposure time image PSF by performing an inversediscrete Fourier transform on the result of the division processing andperforming processing on the result of the inverse discrete Fouriertransform that performs noise removal and identifies a linking componentthat passes through an origin point.

-   (5)

The image processing device according to any one of (1) to (4),

wherein the image correction portion

performs division processing that takes a result of a discrete Fouriertransform that is based on a pixel value for a specific color that hasbeen selected from the long exposure time image and divides it by aresult of a discrete Fourier transform that is based on a pixel valuefor a specific color that has been selected from the short exposure timeimage, and

computes the long exposure time image PSF by performing an inversediscrete Fourier transform on the result of the division processing,performing processing on the result of the inverse discrete Fouriertransform that performs noise removal and identifies a linking closedregion that passes through an origin point, extending the length of aline segment that links the origin point and the center of gravity ofthe closed region to twice its original length, with the origin point atthe center, and defining the extended line segment as the long exposuretime image PSF.

-   (6)

The image processing device any one of (1) to (5),

wherein the image correction portion

computes a PSF that corresponds to the short exposure time image as ashort exposure time image PSF, computing the short exposure time imagePSF by multiplying, by a factor that is equal to two times the ratio ofan exposure time for a second exposure condition to an exposure time fora first exposure condition, the length of a line segment that links anorigin point and the center of gravity of the long exposure time imagePSF, with the origin point at the center.

-   (7)

The image processing device according to any one of (1) to (6),

wherein the image correction portion includes

a first correction amount computation portion that, from a long exposuretime image PSF that is computed based on the long exposure time image,from an initial image that is generated based on the short exposure timeimage, and from the long exposure time image, computes a firstcorrection amount for the initial image,

a second correction amount computation portion that, from a shortexposure time image PSF that is computed based on the short exposuretime image, and from the initial image, computes a second correctionamount for the initial image, and

an addition portion that adds the first correction amount and the secondcorrection amount for the initial image.

-   (8)

The image processing device according to any one of (1) to (7),

wherein the first correction amount computation portion includes

a first estimated image computation portion that computes a firstestimated image that is a result of estimating an image that is similarto the long exposure time image, based on the initial image and the longexposure time image PSF,

a subtraction portion that computes a first difference image that is thedifference between the long exposure time image and the first estimatedimage, and

a first correction amount estimation portion that computes the firstcorrection amount based on the first difference image and the longexposure time image PSF.

-   (9)

The image processing device according to any one of (1) to (8),

wherein the first estimated image computation portion includes

a color-specific PSF computation portion that computes a firstcolor-specific PSF that combines the characteristics of the longexposure time image PSF for each color (phase) of an image capturesurface of the image capture element,

a first convolution computation portion that performs a convolutioncomputation for each color of an object pixel of the initial image usingthe first color-specific PSF, and

a saturation processing portion that takes a results image that has beenoutput from the first convolution computation portion and outputs animage by replacing pixel values that are not less than a value that isequivalent to a saturated pixel value of the image capture element withthe value that is equivalent to the saturated pixel value.

-   (10)

The image processing device according to any one of (1) to (9),

wherein the first correction amount estimation portion includes

an inverse color-specific PSF computation portion that computes a firstinverse color-specific PSF in which the long exposure time image PSF hasbeen inverted symmetrically with respect to a point,

a second convolution computation portion that performs a convolutioncomputation for each color of an object pixel of the first differenceimage using the first inverse color-specific PSF, and

a multiplication portion that performs a correction amount adjustment bymultiplying a correction amount adjustment parameter times a resultsimage that has been output from the second convolution computationportion.

-   (11)

The image processing device according to any one of (1) to (10),

wherein the image correction portion includes

a demosaic processing portion that performs demosaic processing on thelong exposure time image and the short exposure time image, and

the image correction portion generates the corrected image based on ademosaiced image that is a processing result of the demosaic processingportion.

Furthermore, the processing sequence that is explained in thespecification can be implemented by hardware, by software and by aconfiguration that combines hardware and software. In a case where theprocessing is implemented by software, it is possible to install inmemory within a computer that is incorporated into dedicated hardware aprogram in which the processing sequence is encoded and to execute theprogram. It is also possible to install a program in a general-purposecomputer that is capable of performing various types of processing andto execute the program. For example, the program can be installed inadvance in a storage medium. In addition to being installed in acomputer from the storage medium, the program can also be receivedthrough a network, such as a local area network (LAN) or the Internet,and can be installed in a storage medium such as a hard disk or the likethat is built into the computer.

Note that the various types of processing that are described in thisspecification may not only be performed in a temporal sequence as hasbeen described, but may also be performed in parallel or individually,in accordance with the processing capacity of the device that performsthe processing or as needed. Furthermore, the system in thisspecification is not limited to being a configuration that logicallyaggregates a plurality of devices, all of which are contained within thesame housing.

INDUSTRIAL APPLICABILITY

As explained above, according to the example of the present disclosure,a device and a method are achieved that generate an image with reducedblurring and a wide dynamic range based on a single captured image.Specifically, the device includes an image capture element that outputslong exposure time pixel information and short exposure time pixelinformation based on image capture processing under a control thatprovides different exposure times for individual pixels, and alsoincludes an image correction portion that inputs the pixel informationthat the image capture element has output and generates a correctedimage by performing image stabilization and dynamic range expansionprocessing.

The image correction portion generates a long exposure time image inwhich the pixel values have been set on the assumption that all of thepixels have been exposed for a long time and a short exposure time imagein which the pixel values have been set on the assumption that all ofthe pixels have been exposed for a short time, computes a point spreadfunction (a PSF) that corresponds to the long exposure time image as along exposure time image PSF, and generates the corrected image by usingthe short exposure time image, the long exposure time image, and thelong exposure time image PSF. The corrected image is generated as a widedynamic range image that utilizes the pixel information for the longexposure time image and the pixel information for the short exposuretime image. Utilizing the pixel information for the short exposure timeimage, in which there is little blurring, also makes the corrected imagea high quality corrected image in which the blurring is suppressed.

The present disclosure contains subject matter related to that disclosedin Japanese Priority Patent Application JP 2011-102915 filed in theJapan Patent Office on May 2, 2011, the entire content of which ishereby incorporated by reference.

What is claimed is:
 1. An image processing device, comprising: an imagecapture unit configured to generate an initial output image, wherein theimage capture unit comprises a plurality of pixels, and wherein theplurality of pixels include a first group of pixels with a firstexposure time and a second group of pixels with a second exposure time;an interpolation processing unit configured to generate at least a firstinterpolated image based on the first group of pixels with the firstexposure time and a second interpolated image based on the second groupof pixels with the second exposure time; a bit shift unit configured togenerate at least two gradation images, which include a first gradationimage corresponding to the first interpolated image and a secondgradation image corresponding to the second interpolated image; acorrection amount computation unit configured to compute at least twocorrection amounts, wherein the at least two correction amounts includea first correction amount and a second correction amount, and whereinthe first correction amount is computed based, at least in part, on thefirst gradation image and the initial output image, and the secondcorrection amount is computed based, at least in part, on the secondgradation image and the initial output image; and an addition unitconfigured to add the at least two correction amounts to the initialoutput image to produce a corrected output image.
 2. The imageprocessing device of claim 1, further comprising a noise reduction unitconfigured to reduce noise of data associated with the secondinterpolated image to generate the initial output image.
 3. The imageprocessing device of claim 1, wherein the first correction amount isfurther based, at least in part, on a first point spread function (PSF)corresponding to the first interpolated image, and wherein the secondcorrection amount is further based, at least in part, on a second pointspread function (PSF) corresponding to the second interpolated image. 4.The image processing device of claim 1, wherein the image capture unitis configured to control the first and second groups of pixels such thatthe first exposure time and the second exposure time start at differenttimes and end at the same time.
 5. The image processing device of claim1, wherein the interpolation processing unit, to generate the firstinterpolated image, is configured to: generate portions of the firstinterpolated image corresponding to positions of the first group ofpixels based on pixel values of the first group of pixels; generateinterpolated pixel values by interpolation of the pixel values of thefirst group of pixels; and generate portions of the first interpolatedimage corresponding to positions of the second group of pixels based, atleast in part, on the interpolated pixel values.
 6. The image processingdevice of claim 5, wherein interpolation of the pixel values of thefirst group of pixels further comprises interpolation of the pixelvalues of the first group of pixels by filters or linear interpolationof the pixel values of the first group of pixels.
 7. The imageprocessing device of claim 6, wherein generation of the interpolatedpixel values comprises assignment of weights to the pixel values of thefirst group of pixels based, at least in part, on locations of edges inthe first interpolated image.
 8. An image processing method, comprising:generating, with a plurality of pixels, an initial output image, theplurality of pixels including a first group of pixels with a firstexposure time and a second group of pixels with a second exposure time;generating at least a first interpolated image based on the first groupof pixels with the first exposure time and a second interpolated imagebased on the second group of pixels with the second exposure time;generating at least two gradation images by bit shifting at least thefirst and second interpolated images, wherein the at least two gradationimages include a first gradation image corresponding to the firstinterpolated image and a second gradation image corresponding to thesecond interpolated image; determining at least two correction amountsincluding a first correction amount and a second correction amount,wherein the first correction amount is determined based, at least inpart, on the first gradation image and the initial output image, and thesecond correction amount is determined based, at least in part, on thesecond gradation image and the initial output image; and adding the atleast two correction amounts to the initial output image to produce acorrected output image.
 9. The image processing method of claim 8,further comprising: generating the initial output image by reducingnoise of data associated with the second interpolated image.
 10. Theimage processing method of claim 8, wherein the first correction amountis further determined based, at least in part, on a first point spreadfunction (PSF) corresponding to the first interpolated image, andwherein the second correction amount is further determined based, atleast in part, on a second point spread function (PSF) corresponding tothe second interpolated image.
 11. The image processing method of claim8, wherein generating the initial output image comprises controlling thefirst and second groups of pixels such that the first exposure time andthe second exposure time start at different times and end at the sametime.
 12. The image processing method of claim 8, wherein generating thefirst interpolated image based on the first group of pixels with thefirst exposure time comprises: generating portions of the firstinterpolated image corresponding to positions of the first group ofpixels based on pixel values of the first group of pixels; generatinginterpolated pixel values by interpolating the pixel values of the firstgroup of pixels; and generating portions of the first interpolated imagecorresponding to positions of the second group of pixels based, at leastin part, on the interpolated pixel values.
 13. The image processingmethod of claim 12, wherein interpolating the pixel values of the firstgroup of pixels further comprises interpolating the pixel values of thefirst group of pixels by filters or linearly interpolating the pixelvalues of the first group of pixels.
 14. The image processing method ofclaim 13, wherein generating the interpolated pixel values comprisesassigning weights to the pixel values of the first group of pixelsbased, at least in part, on locations of edges in the first interpolatedimage.
 15. A camera, comprising: an image processing device including:an image capture unit configured to generate an initial output image,wherein the image capture unit comprises a plurality of pixels, andwherein the plurality of pixels include a first group of pixels with afirst exposure time and a second group of pixels with a second exposuretime, an interpolation processing unit configured to generate at least afirst interpolated image based on the first group of pixels with thefirst exposure time and a second interpolated image based on the secondgroup of pixels with the second exposure time, a bit shift unitconfigured to generate at least two gradation images, which include afirst gradation image corresponding to the first interpolated image anda second gradation image corresponding to the second interpolated image,a correction amount computation unit configured to compute at least twocorrection amounts, wherein the at least two correction amounts includea first correction amount and a second correction amount, and whereinthe first correction amount is computed based, at least in part, on thefirst gradation image and the initial output image, and the secondcorrection amount is computed based, at least in part, on the secondgradation image and the initial output image, and an addition unitconfigured to add the at least two correction amounts to the initialoutput image to produce a corrected output image; and a display deviceconfigured to display the corrected output image.
 16. The camera ofclaim 15, wherein the camera comprises a digital still camera.
 17. Thecamera of claim 15, wherein the camera comprises a video camera.
 18. Thecamera of claim 15, wherein the image capture unit includes a CMOS imagesensor that comprises the plurality of pixels.